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Abstract 

Our theoretical model first identifies the context in which students experience difficulties with 
complex multiple choice examinations (MCE) and then provides a structured approach to 
facilitate the development ofcritical thinking and metacognitive skills. Based on our model, we 
also designed a brief educational workshop, now in videotaped format. Workshop participants 
were taught that multiple choice (MC) questions assess lower and higher level critical thinking 
skills, and appropriate techniques for study and MC test-taking. To encourage the transfer of 
appropriate learning, participants then processed deeply, practiced, and mastered these mes¬ 
sages through MC test simulations. On anonymous questionnaires, participants reported 
increased knowledge concerning the process and content demands of complex MCE, as well as 
improved confidence and new behavioural skills. 

Resume 

Notre modele theorique definit d'abord le contexte dans lequel les etudiants eprouvent de la 
difficulte a passer un examen a choix multiple complexe (ECM), et ensuite il propose une 
approche structuree pour faciliter le developpement de la pensee critique et des capacites 
metacognitives. Nous avons aussi con^u, a partir de notre modele, un bref atelier d’informa- 
tion, qui est maintenant disponible sous forme de video. Les participants a cet atelier ont appris 
que les questions a choix multiple (CM) evaluent les capacites d’analyse critique inferieures et 
superieures; ils ont egalement appris des techniques precises pour se preparer aux examens a 
choix multiple et pour les passer. Pour faciliter le transfert des connaissances pertinentes, les 
participants ont precede a des simulations d’examens a CM, qui leur ont permis d’assimiler 
profondement les messages, de les appliquer et de les maitriser. Repondant a un questionnaire 
anonyme, les participants ont indique qu’ils parvenaient a mieux comprendre le processus et a 
mieux saisir la matiere de ECM complexes et qu’ils acqueraient davantage de confiance en eux- 
memes et de meilleures habitudes. 


Complex multiple choice examinations (MCE) are a popular form of 
assessment of student knowledge and aptitude across diverse pedagogi¬ 
cal domains ranging from introductory courses to graduate school ad¬ 
mission tests (Aiken, 1987; Cizek, 1994; Skakun, Maguire, & Cook, 1994). 
Despite the need to utilize high level critical thinking skills during study 
and MC test-taking, research shows that high school and beginning 
university students often learn novel material using lower level thinking 
and study skills such as repetition, memorization, and summarization 
(Brown & Campione, 1978; Feldt & Ray, 1989; McDermott, Wood, & 
Willoughby, 1995; Perry, 1981; Thomas & Bain, 1982). Moreover, the 
metacognitive awareness of the need to alter strategies to accommodate 
complex task demands, and the ability to do so, appear to develop slowly 
(Brown & Campione, 1978; Hartley, 1986; McDermott etal., 1995; Perry, 
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1981; Rosenthal, McKnight, Soper, & Baudouin, 1996; Thomas & Bain, 
1982). Our experiences in teaching and counselling at post-secondary 
institutions suggest that the impact of unreadiness for MCE is multi¬ 
faceted, potentially resulting in impaired academic performance, emo¬ 
tional distress, decreased motivation, and premature drop out (also see 
Perry, 1970). 

Although most universities offer generalist study skills programs, we 
note that students often fail to discern the higher level cognitive and 
metacognitive processes that underlie successful performance on MCE 
(Watkins, 1982). Moreover, there is a dearth of published information 
about programs specifically targeted to improve performance on MCE 
through the development of critical thinking skills. To address this gap, 
we carried out a needs assessment of our target group: undergraduate 
students doing poorly on MCE. We questioned students informally dur¬ 
ing one-to-one meetings or small group sessions about their expectations 
and experiences surrounding MCE and their approaches to preparation 
and test-taking. Specifically, we asked students to detail their past experi¬ 
ence with MCE, their expectations for performance, their cognitions 
(both positive and negative) during preparation and test-taking, how 
they prepared for MCE versus essay style exams, and their accuracy in 
predicting exam grades. We determined that many students fail to recog¬ 
nize that well-designed MCE can assess both lower level thinking skills 
(such as the recognition and recall of information) and higher level 
thinking skills (including the ability to apply, synthesize, and integrate 
material). Not surprisingly, these students predominantly used lower 
level study strategies. We further interviewed students who typically did 
well on complex MCE and noted their strategies and cognitions. In 
contrast, we perceived these students to be well equipped cognitively and 
metacognitively to master the course curriculum. In general, these stu¬ 
dents seemed better able to conceptualize and synthesize information, to 
identify important versus less important material, to prioritize study time, 
and to be aware of when they had mastered the material. 

Several themes for success on complex MCE emerged from the inter¬ 
views. First, heightened anxiety and hyperarousal interfere with study 
and test-taking; positive attributions set the stage for success. Second, 
students require a basic knowledge of MC test-taking strategies (e.g., 
dealing with combination items such as “all of the above”; time allocation 
on the test, etc.). Third, students benefit from an understanding of the 
higher level cognitive demands of the MC test process. Often, they 
require some experience with MC test-taking to recognize the complexity 
of the demands. Fourth, to meet the high level demands, students must 
develop mastery level study skills. Finally, repeated practice with feed¬ 
back is necessary to reinforce skill acquisition. Consequently, we devel¬ 
oped a theoretical model to represent the multifaceted process that we 
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suggest underlies successful performance on MCE, and converted it into 
an educational workshop. 

THE MULTIPLE PROCESS MODEL 

Our interviews revealed that many university students perform poorly on 
initial complex MCE and fail to improve performance on subsequent 
MCE, despite investing hours of effort. Without intervention, the poten¬ 
tial exists for negative attributions about self and/or professor, with 
cognitive, affective, and behavioural consequences, resulting in repeated 
negative experiences (Perry, 1970). To alter this pattern, our model 
argues for the necessity of making explicit to students the context in 
which university teaching and evaluation are grounded. That is, students 
must understand that MCE are constructed to assess high level thinking 
skills, thereby drawing upon their cognitive, metacognitive, and affective 
resources throughout the learning cycle. This recognition that low level 
learning strategies are insufficient for MC task demands may prompt the 
development of more suitable attributions and study behaviours. Be¬ 
cause students may require intervention in several domains, our model 
has five facets: Expectations and Anxiety, Test-Taking Strategies and 
Simulated MCE, Teaching Sophisticated Thinking Skills, Mastery Learn¬ 
ing Strategies, and Repeated Practice with Reinforcement. Central to our 
model is the premise that students must be encouraged to deeply process 
information (see Figure 1). 

In developing our model, we also reviewed research from cognitive 
psychology to determine how learners process and synthesize complex 
material (Craik & Lockhart, 1972; Craik & Tulving, 1975; Gibbs, 1981; 
Hartley, 1986; Nickerson, 1988-1989; Thomas & Bain, 1982; Weinstein & 
Mayer, 1986) and examined the literature addressing the content and 
process demands underlying MCE (e.g., Aiken, 1987; Appleby, 1990; 
Benjamin, Cavell, & Shallenberger, 1984; Dolly & Williams, 1986; Fleet, 
Goodchild, & Zajchowski, 1990; Shatz & Best, 1987; Skinner, 1983; Spiers 
& Pihl, 1987). Our review suggests that complex material is best learned 
using high level cognitive and metacognitive strategies that enable stu¬ 
dents to richly encode material, to integrate knowledge via the processes 
of assimilation and accommodation, and then to retrieve material. Mas¬ 
tery of material likely is facilitated by information reduction, reorganiza¬ 
tion, and restructuring (e.g., chunking and hierarchical organization), 
as well as by elaboration and positive imagery. 

DESCRIPTION OF THE WORKSHOP 

To encourage attendance, we advertised a single session (2-hr) workshop 
that promoted student success on MCE through experiential, student- 
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Student Experience 

i. Low level expectations and superficial processing strategies for MCE 

ii. Negative experience on MCE 

iii. Perceived threat of MCE (i.e., anxiety, low confidence, expectation of 
poor performance) 

Interventions 

Facet 1: Identification of Negative Cycle Of MC Experience; Attitude 
Inoculation; Modelling Positive Counterstrategies 
Facet 2: Presentation of MC Test-Taking Strategies; Application through 
Test 1 Simulation 

Facet 3: Analysis of Complex MC; Processing Levels of Critical Thinking 
through Test 1 Scoring 

Facet 4: Teaching Mastery Learning Strategies; SQ3R 

Facet 5: Repeated Practice through MC Test 2 Simulation; Reinforcement 

of New Attitudes, Knowledge, Behaviour through Test 2 Scoring 

Student Outcome 

i. Increased confidence re MCE 

ii. Shift to deeper processing strategies (e.g., SQ3R) 

iii. Increased mastery of complex MCQ 



COGNITIVE 


Figure 1. 

The Multiple Process Model as Represented Through a Workshop Intervention. 
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centred activities. We told students that there would be opportunities to 
apply new knowledge and receive immediate feedback through simu¬ 
lated MC testing. This message attracted undergraduate and graduate 
students from diverse disciplines and programs (e.g., psychology, com¬ 
merce), with approximately 50 students attending each of the first eight 
workshops. Although most were seeking to improve performance on 
MCE in academic courses, some were preparing for graduate admission 
examinations (e.g., GMAT, GRE, LSAT, MCAT). 

Our workshop format consisted of five cumulative sections, corre¬ 
sponding to our theoretical model, designed to encourage cognitive, 
metacognitive, and affective development. We taught participants 
that MCE assess varying levels of critical thinking skills and require 
deep processing. To encourage transfer of appropriate learning (Morris, 
Bransford, & Franks, 1977), we provided opportunities to deeply pro¬ 
cess, practice, and master this message through two simulated MCE. 


Facet 1: Expectations and Anxiety 

Ice breaker: Expectations for the workshop and MCE. We opened by asking 
participants their reasons for attending, past experiences with MCE, and 
strategies used for preparation or test-taking. In support of our model, 
students’ responses reflected many misunderstandings surrounding 
MCE, as well as considerable anxiety. Typically, students did not recog¬ 
nize that MCE comprised more than low level recall and recognition 
items (Watkins, 1982). Moreover, like Thomas & Bain (1982), their study 
strategies mainly encompassed superficial processing of material and 
were uniformly applied regardless of evaluation format (i.e., objective or 
essay style). 

Attitude inoculation: Eliciting and processing “confidence killer” statements. 
We asked participants to share the negative, anxiety-ridden cognitions 
that accompany experiences with MCE. We then exposed them to similar 
defeatist comments gathered from students during prior workshops 
(e.g., “I’m going to throw up!” or “I’m going to miss a box and fail the 
whole test!”). By drawing upon the humor of the exaggerated examples, 
we defused some anticipatory anxiety and identified distorted patterns of 
thinking. Participants’ contributions allowed us to identify the cycle that 
linking negative thoughts, escalation of anxiety, fixation of negative 
beliefs, and negative influence has on MCE outcome. We then modeled 
positive counterstrategies: challenging catastrophic predictions; gener¬ 
ating more coping-focused, realistic, and confidence-enhancing state¬ 
ments; and decreasing anxiety through deep breathing and visualization 
of success (Bums, 1980; Meichenbaum, 1977). 
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Facet 2: Test-Taking Strategies and Simulated MCE 

Basic test-taking strategies for MCE. We distributed a two-part hand¬ 
out compiled from Canadian sources (Fleet et al., 1990; Taylor, Avery, 
Brandow, & Strath, 1988). The first part described strategies to assist test- 
takers in the interpretation of instructions, in the processing of question 
stem qualifiers and absolutes, and in the maintenance of emotional 
equilibrium. The second part provided a 10-step answering system de¬ 
signed to encourage deeper processing of the MC question (Craik & 
Lockhart, 1972; Craik 8c Tulving, 1975) and prediction of the possible 
answer. We then recommended that participants apply this system dur¬ 
ing the first test-taking simulation. 

First test-taking simulation: Mount St. Vincent University reading and MC test. 
Participants studied (4 min) a brief passage entitled The Mission State¬ 
ment of Mount St. Vincent University (1993) and completed (6 min) a 
corresponding six-item MCE (Wetmore & Bowering, 1995). Participants 
were not informed that the MCE included questions of increasing cogni¬ 
tive complexity. 

Facet 3: Teaching Sophisticated Thinking Skills 

Scoring the first test: Critical analysis of levels of complexity of MC questions. Our 
model suggests that successful performance derives, in part, from the 
understanding that complex MCE are constructed to assess a range of 
critical thinking skills. During the scoring of the first test, we introduce 
these skills as the “Levels of Multiple Choice” (adapted with permission 
from Appleby, 1990). The least sophisticated skill (i.e., Level 1) requires 
simple recall/recognition of information. Level 2 involves memorization of a 
set of characteristics, with assigning of priority of importance to elements. 
Level 3 requires sequencing events in time or in magnitude (e.g., chrono¬ 
logical relationships). Level 4 requires application of knowledge (e.g., 
diagnosis of disease). Level 5 involves the identification of similarities 
and differences and the conceptualization of relations (e.g., analogies). Level 6, 
the most sophisticated skill, requires the synthesis of information 
(e.g., integrating knowledge, drawing inferences). In processing the test, 
we linked each of the six questions to its corresponding level of MC, 
elaborated on the type of thinking skills required for success, and di- 
agrammatically illustrated the cognitive response process. Thus, our 
theoretical synthesis of Appleby (1990), Bloom (1956), and Craik (Craik 
& Lockhart, 1972; Craik 8c Tulving, 1975) was extended into a practical 
teaching application. After scoring the test, participants said that they 
had been preparing as if MCE assessed only recognition and recall (e.g., 
using memorization and rote learning). Not surprisingly, participants 
now had different attributions for their errors on questions from Levels 
4-6. 
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Facet 4: Mastery Learning Strategies 

Strategies for deeper processing: SQ3R. Once participants realized the cogni¬ 
tive complexity demanded by MCE, they were primed to modify their 
study behaviour. We then introduced SQ3R (i.e., Survey, Question, Re ad, 
Recite, and .Review) (Robinson, 1961) as a method to enhance reading 
and retention, identification and processing of concepts, and ultimately 
the integration and synthesis of course content. Participants were in¬ 
structed to skim the passage, formulate general questions, read for 
identification and comprehension of concepts, attempt to answer their 
own questions, summarize material, and anticipate MCQ. These activ¬ 
ities contribute to students’ ability to analyze their depth of processing of 
the material, an impdrtant metacognitive skill. 

Facet 5: Repeated Practice with Reinforcement 

Second test-taking simulation: Mood disorder reading and MC test. Participants 
studied (6-min) a one-page passage entitled Mood Disorders (Wetmore 
& Bowering, 1995) and completed (10-min) a corresponding 12-item 
MCE (Wetmore & Bowering, 1995). Participants were forewarned that 
the test would emphasize questions from Levels 4—6, and urged to apply 
their new knowledge concerning SQ3R, MC test-taking strategies, and 
the critical thinking skills required for success. We also asked students to 
monitor how these new strategies increased their depth of processing. 

Scoring the second test: Reinforcement of new attitude, knowledge, and behaviour. 
Again, we scored each test item referring to the corresponding level of 
MC. Even participants who had performed very poorly on the first test 
generally improved their scores. Many articulated that their learning 
strategies had shifted (e.g., from re-reading and rote memorization to 
chunking and identification of main points/concepts). They expressed 
newfound confidence in their ability to handle complex MC (e.g., the 
questions were no longer viewed as “tricky” and “out to get them” but, 
rather, required the use of complex processes to select the correct 
answer). We interpreted their comments as reflecting deeper processing 
of the material and development of critical thinking skills. 

PROGRAM EVALUATION: EFFECTIVENESS OF WORKSHOP INTERVENTION 

After summarizing the key points of our workshop, we measured partici¬ 
pants’ satisfaction via anonymous questionnaires completed at the end 
of the session. They were asked to identify the most and least helpful 
components of the workshop, and using a 5-point scale (1 = Poor, 5 = 
Excellent) to rate the workshop’s quality, amount of information, useful¬ 
ness of information, method of presentation, and overall workshop rating. 

To accommodate the demand for the workshop (now over 400 partici¬ 
pants), the live presentation had been transferred onto a professionally 
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edited videotape of approximately 90 minutes (Wetmore & Bowering, 
1994). We selected at random four workshops presented during 1993 to 
1996, two of which were the videotaped version and two the live version, 
and analyzed the feedback questionnaires (n = 110). (Note that three 
respondents skipped one question on the form.) One-way ANOVA’s 
indicated that the four versions did not differ significantly in quality 
[7^(3,105) = 0.58, /? = .63), amount [7^3,106) = 0.46, p= .71), usefulness 
[7 r (3,106) = 1.90,/? =.13), or overall workshop rating [7^3,100) = 1.14,/? = 
0.34). Although the method of presentation was significant [7 r (3,106) = 
3.57, /?= .0165), Scheffe tests failed to reveal the source of the difference 
( p’s > .05). With a mean overall rating of 4.5 on a 5-point scale, evalua¬ 
tions were consistently favorable for both presentation types (see Table 
1). Clearly, the video format is both cost effective and has ensured 
consistency over time. 

The questionnaire for workshop 4 added a new category: ratings of 
self-confidence regarding MCE pre- and post-workshop. Participants 
reported that their confidence improved significantly from before (M = 
2.2) to after (M = 3.9) the workshop [7(14) = -10.5, p < .0001]. 

Finally, an open-ended section on the questionnaire allowed com¬ 
ments on changes in knowledge and attitudes. The most frequently 
indicated helpful components were the levels of multiple choice, oppor¬ 
tunities for application through repeated testing, and SQ3R. Many stu¬ 
dents wrote that all components were helpful and did not answer the 
question concerning the least helpful component. For example, one 
participant wrote “The workshop was very helpful and the information 
will be extremely helpful in all areas of study. Thank you sooo much!.” Also 
supporting the improved ratings of self-confidence, and thus the value of 
attitude inoculation, were responses that revoked previously held nega¬ 
tive beliefs and indicated increased motivation. 

In summary, the response to our workshop has been overwhelmingly 
positive, including requests for the videotape from other universities. 
Students’ comments (over a 5-year period) indicate that the workshop 
has identified many of their difficulties with MCE and provided a forum 
to rectify these problems. Underaddressed, however, is whether there is 
transfer of training from the workshop to performance on MCE in 
university courses. Difficult to partial out is the influence of variables 
such as student’s intellectual ability, lifestyle commitments, and class¬ 
room climate. Nonetheless, unsolicited feedback from participants and 
faculty confirms that the workshop has improved performance on MCE. 
It would be valuable to know which components are particularly effective 
in mediating the improvement. (We suspect that the key components are 
recognition of the levels of complexity being tested and acquisition of 
mastery level study skills.) Additionally, we queried whether there will be 
long-term maintenance of the new knowledge, attitudes, and behaviours. 
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TABLE 1 

Summary Ratings as a Function of Workshop Presentation 


Question 

l 

Video 

n =26 

2 

Live 
n = 47 

Workshop 

3 

Live 

n =22 

4 

Video 

n = 15 

Overall Rating 

M 

4.4 

4.6 

4.4 

4.5 

SD 

0.6 

0.5 

0.1 

0.5 

R 

3-5 

3-5 

2-5 

4-5 

Quality 

M 

4.5 

4.5 

4.5 

4.3 

SD 

0.5 

0.6 

0.6 

0.6 

R 

4-5 

3-5 

3-5 

3-5 

Amount 

M 

4.4 

4.6 

4.4 

4.5 

SD 

0.5 

0.7 

0.2 

0.5 

R 

3-5 

3-5 

3-5 

4-5 

Usefulness 

M 

4.3 

4.6 

4.5 

4.5 

SD 

0.7 

0.6 

0.2 

0.5 

R 

3-5 

3-5 

3-5 

4-5 

Method 

M 

4.1 

4.6 

4.7 

4.1 

SD 

0.8 

0.6 

0.7 

0.8 

R 

2-5 

3-5 

2-5 

2-5 


Note : A higher score (5 point scale: 1 = Poor, 5 = Excellent) indicates a more positive evaluation. 


Informal follow-up even a year later of some participants suggests that 
they have persisted in application of our techniques even though con¬ 
fronted by the competing demands of many university courses and the 
limited time for studying (also see Watkins, 1982). Long-term stability 
may not be surprising given that our workshop addressed some obstacles 
to behaviour change, including awareness of relevant information (e.g., 
levels of multiple choice), inoculation, and repeated modeling of appro¬ 
priate behaviours. 


CONCLUSIONS 

Our Multiple Process model and workshop intervention address stu¬ 
dents’ unreadiness for MCE through exposure to deeper processing 
strategies and the training of critical thinking skills. Questionnaire data 
suggest that even a brief (2-hour) intervention can increase participants’ 
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awareness of the complexity of MCE, improve self-confidence, and pro¬ 
mote deeper processing skills. Given the popularity of MCE, students 
require comprehensive instruction about both its content and process 
demands. These findings have implications for counselling psychologists 
and teaching faculty: Learning how to prepare effectively for complex 
MCE may be linked to mastery of the course curriculum and the develop¬ 
ment of critical cognitive and metacognitive skills—primary objectives of 
post-secondary institutions. 
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